Digitizer using position-unique optical signals

ABSTRACT

The present invention provides systems and methods of using a stylus that houses optics and a detector capable of receiving optical signals that are combined with a displayed image. Stylus position determination is made by analyzing received optical signals.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 61/507,669, filed Jul. 14, 2011, the disclosure of whichis incorporated by reference herein in its entirety.

BACKGROUND

Pointing devices of various types are used in almost every computerapplication today. Some of the pointing devices that have been developedand popularized include:

-   -   (1) a mouse;    -   (2) a trackball;    -   (3) a transparent-touch screen that overlays the display        including resistive and capacitive types;    -   (4) pressure, capacitive, resistive, or thermal sensitive tablet        separate from the display;    -   (5) beam-breaking detectors surrounding the display;    -   (6) a light pen based on detection of the raster scan timing of        the phosphor refresh beam;    -   (7) a stylus that incorporates pressure transducers; and    -   (8) a pen using ultrasonic, stereo tactic or radio frequency        triangulation methods.

The various devices developed to position a pointer on a computer screenoperate in conjunction with the application software on the computerusing an appropriate software driver. Most such drivers try to emulatethe conventional X/Y roller mouse. Typically, only positionalinformation is fed to the application software that controls theposition of the pointer on the screen.

Pens may offer increased fidelity of user interaction with computersystems. However, depending on implementation, the resolution of penposition determination may be limited to that of the touch-sensitivesensor. Often, touch screens and tablets have a touch resolution that iscoarser than the screen resolution of the computer for which they areacting as an input device.

SUMMARY

A digitizer system wherein a backlight used in an electronic display isconfigured to emit signals that can be sensed by a stylus. The styluscan determine, from the emitted signals, its location proximate theelectronic display. The emitted signals may, in one embodiment, comprisehuman-visible light that is modulated at a frequency that isundetectable, or at least minimally noticeable, by a human user. In oneembodiment, the stylus may determine its own position and report, viaradio, associated coordinates back to a host computer, which may thenupdate information on the display based on the coordinate information. Aplurality of styli may be used in conjunction with the system sodescribed, and the system may scale with relative ease to accommodateadditional styli.

In one embodiment, a digitizer system is described, comprising a displaydevice having a display area comprising a plurality of pixels; abacklight coupled to the display device; an optical stylus comprising anoptical sensor coupled to a stylus body; an optical stylus processorcommunicatively coupled to the optical sensor;

wherein the backlight is configured to emit at least a first and asecond optical positional signal at a first and a second position; andwherein the optical stylus receives the optical positional signals, andthe optical stylus processor generates location-related positionalsignals based on the received optical positional signals.

In another embodiment, a display system is described, comprising adisplay device having a display area comprising a plurality of pixels; abacklight coupled to the display device; an optical stylus comprising anoptical sensor coupled to a stylus body; an optical stylus processorcommunicatively coupled to the optical sensor; wherein the backlight isconfigured to emit at least a first and a second optical positionalsignal at a first and a second position; and wherein the optical stylusreceives the optical positional signals, and the optical stylusprocessor generates location-related positional signals based on thereceived optical positional signals.

Related methods, systems, and articles are also discussed. These andother aspects of the present application will be apparent from thedetailed description below. In no event, however, should the abovesummaries be construed as limitations on the claimed subject matter,which subject matter is defined solely by the attached claims, as may beamended during prosecution.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic of an LCD based digitizer system;

FIG. 1B is a schematic of an emissive display based digitizer system;

FIG. 1C is a schematic of a projection display based digitizer system;

FIG. 2 is a schematic of a cross-section of an optical stylus;

FIG. 3 is a simplified drawing of a backlight employing position-uniqueoptical signals;

FIG. 4 is a simplified drawing of a backlight employing position-uniqueoptical signals;

FIG. 5A is a time-axis chart showing representations of position-uniqueoptical signals at various points on a display;

FIG. 5B is another time-axis chart showing representations ofposition-unique optical signals at various points on a display;

FIG. 6 is another time-axis chart showing representation ofposition-unique optical signals at various points on a display;

FIG. 7 is a simplified drawing of a backlight employing position-uniqueoptical signals; and,

FIG. 8 is a simplified drawing of a full-array backlight employingposition-unique optical signals.

DETAILED DESCRIPTION

Movement of an optical stylus relative to a display may be measured bysensing position-unique optical signals provided, for example, by abacklight associated with the display. For example, light-emittingelements of an liquid crystal display (LCD)-type display device may beconfigured to emit position-related signals of light, possibly in atime-varying (modulated) manner, from different positions of thedisplay, such that the combination of two (or more) such positionsignals at any given point on the display may be sensed by a stylus andused to determine the position of the stylus relative to, or on, thedisplay device.

FIG. 1A is a schematic of a digitizer system of embodiment A. Display110 is an electronically addressable light shuttering display, such asan LCD. In LCD embodiments, display 110 includes a backlight 112, whichprovides human visible light through LCD panel 111. Backlight 112includes at least two, and possible more, emitters which emit modulatedlight signals into components of display 110. The positional-modulatedemitters may also emit visible light through the LCD, or they may emitnon-visible light. The positional modulation of emitters, and theassociated modulation of light signals, minimally affect the visibleimage being displayed by the electronically addressable display.

Optical stylus 120 includes an optical sensor coupled to a stylus body.The optical sensor receives the modulated light signals, which areprocessed by an optical stylus processor that may also be includedwithin optical stylus 120, to determine the position of the opticalsensor upon display 110. Communications link 124 communicatively couplesstylus 120 to computing device 130, which may be a notebook or tabletcomputer, which is in turn communicatively coupled to display 110.Communications link 124 may comprise several conductors, or a radio linksuch as that implementing the wireless communications protocol referredto as Bluetooth™. Optical stylus processor includes electronicsnecessary to provide electrical signals indicative of the sensed signalscomputing device 130. Communications link 135 communicatively couplesdisplay device 110 with computing device 130. This coupling may be by aplurality of conductors or may be by radio. Communications link 136communicatively couples backlight 112 with computing device 130 for thepurpose of controlling backlight emitters. While computing device 130 isshown separate from display device 110, other integral combinations arepossible, where both the display device and the computer are within thesame chassis (as would be the case, for example, with tablet-typecomputers, all-in-ones, and laptops).

FIG. 1B is a schematic of a digitizer system 140 of embodiment B.Emissive display 141 is an electronically addressable visible lightemitting display that is at least partially transparent or translucentat some wavelengths of light, for example infra-red. Examples of thisdisplay type include plasma displays and transparent organic lightemitting diode (OLED) displays. Backlight 142, provides non-visiblelight through emissive display panel 141. Backlight 142 includes atleast two, and possible more, emitters, which emit modulated lightsignals through display 141. The emitters, and the associated modulationof light signals, are preferably non-visible so they minimally affectthe visible image being displayed by emissive display 141.

FIG. 1C is a schematic of a digitizer system 150 of embodiment C.Display surface 151 is a light diffusing surface that is transparent ortranslucent at some non-visible wavelengths of light, and may betranslucent at visible wavelengths of light. Examples of this displaytype include rear projection displays and front projection displays. Ina front projection embodiment, visible light 158 is reflected fromdisplay surface 151. In a rear projection embodiment, visible light 159is passes through backlight 152 and is diffused by display surface 151.Backlight 152 provides position-unique non-visible light (e.g. IR)through display surface 151. Backlight 152 includes at least two, andpossible more, emitters, which emit modulated light signals throughdisplay surface 151. The emitters, and the associated modulation oflight signals, minimally affect the visible image being displayed on thedisplay surface. Backlight 152 is depicted with a flat cross section,though back lighting may also be projected onto surface 151 from adistance.

FIG. 2 depicts a cross sectional view of an optical stylus 50 that canbe used in the present invention. Lens 52 focuses light 55 from adisplay through aperture 53 onto signal detector 54, which is preferablya photodiode, phototransistor, or may be a charge coupled device (CCD)array, for example (these sensing components referred to collectively asan optical sensor).

Stylus signal detector 54 is communicatively coupled to optical signalprocessor 58. Optical signal processor receives signals from opticaldetector 54 through interconnection 56. In embodiments where detector 54is a photodiode or phototransistor, processor 58 measures a signalindicating the intensity of display light 55. In embodiments wheredetector 54 is a 2D image detector such as a CCD, processor 58 measuresa two dimensional map of intensities of display light 55.

In some embodiments, stylus 50 may incorporate a photodiode orphototransistor and also a CCD. For example, a photodiode may detectposition-modulated IR light while a CCD views the visible pixels of thedisplay.

The stylus 50 can have a long focal length so it remains focused on adisplay surface over ranges including when the stylus is contacting thefront surface up to when the stylus is several meters from the displaysurface, for example. Stylus body 51 supports signal detector 54, lens52, and processor 58. Stylus 50 may include optional components such asa tip switch or barrel switch, not shown.

FIG. 3 is a drawing of a backlight 300 having light guide 303 and fouroptical signal emitters, positioned proximate to each corner of thelight guide. Light guide 303 is a component commonly used in LCDs. Ittypically comprises one or more plates of transparent or translucentmaterial. Typical light guides are described in US Patent ApplicationNo. 20100014027 and U.S. Pat. Nos. 7,532,800 and 7,699,516. A lightguide may further include a light reflecting rear surface, and lightdirecting films. Examples of light directing films include brightnessenhancement films sold by 3M Corporation, St. Paul, Minn., under thetrade names Vikuiti BEF I, BEF II, and BEF III. Backlight 300 would bepositioned behind a display, in one embodiment. Positioning the lightemitters around the perimeter of the display is referred to as edgelighting. Individual positional emitters provide optical signals thatmay be modulated with various pulse sequences, creating light signalsthat may be resolved to define unique positions on the display. Thebacklight shown in FIG. 3 has four emitters 310A-310D that provideoptical positional signals, positioned at each of the corners. In someembodiments, emitters may additionally provide the visible light for thedisplay, in displays that require backlighting, such as an LCD display.In other embodiments, positional emitters may be combines withadditional lighting elements included in the edge lighting that are notproviding optical positional signals. Additionally, while four emittersare shown in FIG. 3, other embodiments having for example only twoemitters are possible.

For ease of representation, only representations of signals emitted fromupper right (“UR”) emitter 310A are shown in FIG. 3. Emitter 310A, inthis embodiment, comprises an LED, providing light using light signalsmodulated at a frequency specific to emitter 310A. Other emitters 310B,310C, and 310D, corresponding to locations Lower Right (LR), Lower Left(LL) and Upper Left (UL) would function similarly, but with differentmodulation waveforms or phase from one another. Light guide 303 providesdistribution of the light signals from emitters 310A-310D across thearea of light guide 303, and out of light guide 303 in a directiontoward the user. Emitters, if integral to the backlight scheme of, forexample, an LCD display, are typically located behind the display (fromthe perspective of a user who is located in front of the display), tothusly provide light through the individual pixels of said display.However, other configurations are possible. For example, in embodimentsB and C above where the emitters do not additionally provide forbacklighting, a separate substrate receiving signals provided by theemitters could be located on top (i.e., user-side) of the display, as adigitizer overlay component.

Optical positional signals 320 are represented emanating from emitter310A. Signals emanating from other emitters may have differentwavelengths or they may (and would typically) be of the same wavelength(color) light, but they may have unique modulation wave shape.

FIG. 4 is a drawing of backlight 301 having four optical signalemitters, positioned proximate to each corner of a rectangular display.It is similar to backlight 300 (FIG. 3), but additionally includesrepresentation of optical positional signals associated with each of theemitters 310A-310D. FIG. 4 shows point A and point B, which are referredto in relation to subsequent figures.

FIG. 5 a is a graph showing light signals provided by emitters310A-310D. Emitter UL (310D) is shown having a first waveformmodulation; emitter UR (310A) is shown having a second waveformmodulation; emitter LL (310C) is shown having a third waveformmodulation; and emitter LR (310B) is shown having a fourth waveformmodulation. All four emitters are on for four clock periods duringsynchronization period between t6 and t10. This provides a referencelevel for measuring proportions of light, and it provides asynchronizing signal that an untethered stylus can use as a timereference to determine when each light source is emitting. Arepresentation of the summation of the four waveforms at various timesis shown at the bottom of the graph, in one case associated with pointA, and in another case associated with point B. An optical sensor (as inan optical stylus) would detect the sum of the signals emitted from thefour emitters in proportions corresponding to the location of theoptical sensor on the display. For example, an optical sensor positionedat point A (referring to FIG. 4) would detect equal amounts of lightfrom each emitter. But a sensor positioned at point B would detect morelight from source UR, followed by LR, UL, and LL. This phenomenon meansa ratiometric approach may be used to determine the position of theoptical sensor based on the relative magnitudes of the received signal,which changes as a function of how near they are to particular emitters.

For example with reference to FIG. 5 a, signal B (corresponding to lightmeasured at point B) is constant and maximum during the sync period fromt6 to t10, then each of the four emitters UL, UR, LL, then LR is pulsedoff in sequence between time t10 and time t17. As each emitter is pulsedoff, there is a corresponding reduction in light received at point B.The changes in light are indicated by the magnitudes of pulses P_(UL),P_(UR), P_(LL), and P_(LR) in FIG. 5.

In each case, the pulse magnitude (deviation from the P_(SYNC) level) isproportional to the relative contribution of light from each emitter,which is inversely proportional to the relative distance from eachemitter.

Signal A (corresponding to light measured at point A) shows four pulsesbetween t10 and t17, and all pulses are equal in magnitude. Thisindicates equal light received from the four emitters, which indicatesthat point A is equally distant from each of the emitters UL, UR, LL,then LR.

White LEDs may be used as emitters 310A-310D. Given the modulation shownin FIG. 5 a, the LEDs have a 91% duty cycle so they contribute to theuser-visible light emitted through, for example, an LCD, as well asproviding optical positional signals used to determine the position of astylus proximate the display. Infra-red (IR) LEDs may also or instead beused as emitters 310A-310D, which may be preferable for Embodiments A,B, and C. IR emitters used with Embodiment A may, in some embodiments,have a wavelength greater than 900 nM. Longer wavelength lightpenetrates LCD display pixels in the “on” state or the “off” state, soposition signals may be measured relatively independent of the imagedisplayed on the LCD. For example, LEDs that emit 950 nM IR are readilyavailable.

In embodiments using non-visible light emitting LEDs to generateposition-unique signals, the duty cycle of the LED would preferably bereduced to save power, because such IR LEDs do not contribute touser-visible light emitted from the display. FIG. 5 b shows a graphillustrating representative signals from such an embodiment. FIG. 5 b issimilar to 5 a except the duty cycle of emitters UL, UR, LL, and LRreversed to 10% or less, rather than 91%.

In addition to locating a stylus relative to the surface of a singledisplay, variations in pulse codes of FIG. 5A may be used todiscriminate one display from another. For example, in a system with twoor more displays, the stylus may detect which display is within itsfield of view, and communication from the stylus to its host computersystem includes a display ID and corresponding multi-displaycoordinates. Other functions may include limiting a stylus to operatewith one display but not with another.

Pulse codes of multiple displays may be made unique from one another ina variety of ways. For example, one display backlight may use IR pulsesas shown in FIG. 5A while another display's backlight emits the samewaveform, but at a different fundamental frequency, (the time period ofa cycle from t6 to t17 is different). Another display may bedifferentiated by changing only the time period of the sync pulseP_(SYNC) (from t6 to t10). Another display may be differentiated bychanging the shape of pulses P_(UL), P_(UR), P_(LL), and P_(LR), forexample each of pulses P_(UL), P_(UR), P_(LL), and P_(LR), may comprisetwo pulses rather than one, as is shown in the UL waveform of FIG. 6, orthree pulses rather than one, as is shown in the LL waveform of FIG. 6.Additional waveform variations may be used separately or in combinationto generate display-unique light signals.

FIG. 6 shows a graph illustrating representative signals from anotheralternative embodiment. Rather than equal shaped, equally spaced emitterpulses with different phases, the pulses shown in FIG. 6 have variousshapes, which help distinguish which emitter is contributing eachportion of a signal received by an optical sensor.

It should be realized that the optical positional signals represented aswaveforms in FIGS. 5A and 5B and 6 may be used in various combinationsand any of the signals may be used with any wavelength light sources.Also, other modulation methods may be used, including amplitudemodulation, frequency or phase modulation, variations of pulse coding,or sine waves with fixed frequency and amplitude or varying frequencyand/or amplitude, and variations in wavelength of light emitted fromdifferent emitters. Modulated signals provided by emitters arepreferably imperceptible or minimally perceptible to the human eye, dueto the frequencies of modulation or to the wavelength of light that ismodulated.

FIG. 7 is a drawing of backlight 30 including ten optical signalemitters, positioned along the top and bottom edge of light guide 41.This is a simplified representation of an LCD backlight: in practice LCDbacklights may have many more such emitters. This basic layout, however,is typical of backlights using LED edge lighting. Rays representinglight from emitters are shown for illustration purposes. Optical signalemitters 31-40 may be used to locate a stylus if light from at leastsome of the emitters is modulated differently than others. For example,emitters 31-40 may all emit the same color and average brightness inorder to perform their respective backlighting function, butadditionally they may be modulated with signals that result in opticalpositional signals that provide for position-unique signals that may besensed at any position on the display. In one embodiment, each of theemitters 31-40 may modulate its respective optical positional signalsuniquely. Alternatively, some of the emitters may modulate the same asothers, provided that the sum of signals at every location on thesurface of backlight 30 comprises a unique combination from the tenemitters 31-40. Further emitters 31 and 40 overlap minimally so they mayemit identically modulated light. Emitters 35 and 36 may also emitidentically modulated light because their optical positional signals maybe combined with optical positional signals from other emitters suchthat the total emitted light at any given point on backlight 30 will beunique.

FIG. 8 shows full array backlight 80 similar to those used with some LCDdisplays. Backlight 80 is divided into six areas, A, B, C, D, E, and F.Emitters 82 each provide unique modulation of the light they provide,similar to other embodiments described herein.

Rather than white light LEDs, a further embodiment uses LED lasers as anedge light. The laser beam may be scanned across the light guide of anLCD display, refracting at grid lines in the light guide so its lightemits through the visible display at predetermined places as the laserbeam is scanned. The refracted portion of the laser beam may be detectedby the optical sensor.

Unless otherwise indicated, all numbers expressing quantities,measurement of properties, and so forth used in the specification andclaims are to be understood as being modified by the term “about”.Accordingly, unless indicated to the contrary, the numerical parametersset forth in the specification and claims are approximations that canvary depending on the desired properties sought to be obtained by thoseskilled in the art utilizing the teachings of the present application.Not as an attempt to limit the application of the doctrine ofequivalents to the scope of the claims, each numerical parameter shouldat least be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof the invention are approximations, to the extent any numerical valuesare set forth in specific examples described herein, they are reportedas precisely as reasonably possible. Any numerical value, however, maywell contain errors associated with testing or measurement limitations.

Various modifications and alterations of this invention will be apparentto those skilled in the art without departing from the spirit and scopeof this invention, and it should be understood that this invention isnot limited to the illustrative embodiments set forth herein. Forexample, the reader should assume that features of one disclosedembodiment can also be applied to all other disclosed embodiments unlessotherwise indicated. It should also be understood that all U.S. patents,patent application publications, and other patent and non-patentdocuments referred to herein are incorporated by reference, to theextent they do not contradict the foregoing disclosure.

1. A digitizer system comprising: a display device having a display areacomprising a plurality of pixels; a backlight coupled to the displaydevice; an optical stylus comprising an optical sensor coupled to astylus body; an optical stylus processor communicatively coupled to theoptical sensor; wherein the backlight is configured to emit at least afirst and a second optical positional signal at a first and a secondposition; and wherein the optical stylus receives the optical positionalsignals, and the optical stylus processor generates location-relatedpositional signals based on the received optical positional signals. 2.The digitizer system of claim 1, wherein the location-related positionalsignals comprise signals indicative of the coordinates of the opticalstylus on the display area.
 3. The digitizer system of claim 1, whereinthe optical stylus processor is within a personal computing device. 4.The digitizer system of claim 1, wherein the optical stylus processor iswithin the stylus body.
 5. The digitizer system of claim 1, wherein theoptical stylus processor is also a general purpose processor used in apersonal computing device.
 6. The digitizer system of claim 1, furthercomprising a host processor, which receives the location-relatedpositional signals and provides signals to the display device causingthe display device to update its display area based on thelocation-related positional signals.
 7. The digitizer system of claim 6,wherein the optical stylus additionally comprises a stylus radio forcommunicating location-related positional signals.
 8. The digitizersystem of claim 7, additionally comprising a host radio communicativelycoupled to the stylus radio.
 9. The digitizer system of claim 8, whereinthe host radio and the stylus radio are communicatively coupled using awireless protocol.
 10. The digitizer system of claim 9, wherein thewireless protocol implements the Bluetooth standard.
 11. The digitizersystem of claim 1, wherein the first and second optical positionalsignals have a first and a second modulation respectively.
 12. Thedigitizer system of claim 1, wherein the first and second opticalpositional signals are infra-red signals.
 13. The digitizer system ofclaim 1, wherein the first and second modulation frequency is 30 Hz ormore.
 14. The digitizer system of claim 1, wherein the first and secondmodulation frequency is 100 Hz or more.
 15. The digitizer system ofclaim 1, wherein the first and second modulation frequency is 500 Hz ormore.
 16. The digitizer system of claim 1, wherein the first and secondoptical positional signals have first and second waveforms.
 17. Thedigitizer system of claim 1, wherein the first and the second positionare along the edge of the display device.
 18. The digitizer system ofclaim 1, wherein the backlight comprises a plurality of light emittingdiodes.
 19. The digitizer system of claim 18, wherein the light emittingdiodes are arranged in a matrix.
 20. The digitizer system of claim 1,wherein the optical positional signals are imperceptible to the humaneye.
 21. The digitizer system of claim 1, wherein the optical positionalsignals have wavelengths greater than 900 nanometers.
 22. The digitizersystem of claim 1, further comprising a second optical stylus comprisingan optical sensor coupled to a stylus body, the second optical styluscommunicatively coupled to a the optical stylus processor or a secondoptical stylus processor, and wherein the second optical stylus receivesthe optical positional signals, and the optical stylus processorcommunicatively coupled to the second optical stylus generateslocation-related positional signals based on the received opticalpositional signals.
 23. The digitizer system of claim 22, furthercomprising a third optical stylus comprising an optical sensor coupledto a stylus body, the third optical stylus communicatively coupled tothe optical stylus processor or a third optical stylus processor, andwherein the third optical stylus receives the optical positionalsignals, and the optical stylus processor communicatively coupled to thethird optical stylus generates location-related positional signals basedon the received optical positional signals.
 24. A digitizer systemcomprising: a display device configured to display an image in a displayarea; a light guide positioned proximate to the display device; at leasta first and a second light sources emitting at least a first and asecond optical signal into the light guide; an optical stylus comprisingan optical sensor coupled to a stylus body; an optical stylus processorcommunicatively coupled to the optical sensor; wherein the light guideis configured to mix the first and second optical signals with thedisplay image such that positions on the display area are associatedwith a position-unique combination of the first and second opticalsignals; and wherein the optical stylus receives the position-uniquecombination of the first and second optical signals, and the opticalstylus processor generates location-related positional signals based onthe received position-unique combination.
 25. The digitizer system ofclaim 24, wherein the light display device is a projector and thedisplay area is a screen.
 26. The digitizer system of claim 25, whereinthe light guide is positioned between the projector and the screen. 27.The digitizer system of claim 24, wherein the light guide is co-planarwith the display device.
 28. A display system comprising: a displaydevice having a display area comprising a plurality of pixels; abacklight coupled to the display device; an optical stylus comprising anoptical sensor coupled to a stylus body; an optical stylus processorcommunicatively coupled to the optical sensor; wherein the backlight isconfigured to emit at least a first and a second optical positionalsignal at a first and a second position; and wherein the optical stylusreceives the optical positional signals, and the optical stylusprocessor generates location-related positional signals based on thereceived optical positional signals.
 29. A display system comprising: adisplay device configured to display an image in a display area; a lightguide positioned proximate to the display device; at least a first and asecond light sources emitting at least a first and a second opticalsignal into the light guide; wherein the light guide is configured tomix the first and second optical signals with the display image suchthat positions on the display area are associated with a position-uniquecombination of the first and second optical signals.
 30. A displaysystem of claim 24, wherein the optical signals comprise infra-redlight.
 31. A display system of claim 24, wherein the optical signalscomprise infra-red light with wavelengths greater than 900 nanometers.